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CHAPTER I. 
INTRODUCTION 




With the increasing pressure to expand access to information and the growing 
availability of full-text and full-image journal databases, librarians must now decide 
whether their libraries will provide their patrons access to these information services. 
When it is decided that a library will acquire or subscribe to these types of databases it is 
often done so in cooperative efforts with other libraries in order to reduce costs. In fact 
some librarians from these library networks are even using their collective bargaining 
power to persuade reluctant journal publishers to start releasing electronic versions of 
their products. However, the dilemma remains as to which journals will most effectively 
meet their patron’s demand for expanded access to information. Given the considerable 
cost of these databases, it is often only feasible to purchase a select few of these journals 
with the assumption that only the “best” journals are to be considered. The best journals 
could be defined as those that offer the greatest depth of coverage and the most reliable 
research in a specific field of study. With this in mind, an obvious question presents 
itself, how do librarians choose the best journals? 

One method librarians have used to help make journal subscription and other 
acquisition decisions is citation analysis. Citation analysis is based on the convention 
that when researchers write articles, they typically cite work that has contributed in some 
way to their own work. In this manner, citations can offer a kind of historical blueprint 
for current research. Citation analysis offers a quantitative means to search for patterns 
in these citations and can allow observations to be made about the progression of research 
in a particular field. 

The Social Sciences Citation Index (SSCI) and its supplement the Journal 
Citation Reports (JCR), published by the Institute for Scientific Information (ISI), 
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provide the bulk of citation data for the social sciences. Eugene Garfield, ISI founder, 
writes that, “citations are an important indicator of how frequently current researchers are 
using the journal literature. By tabulating and aggregating citations, the JCR offers a 
unique perspective for journal evaluation and comparison .” 1 One common assumption 
that has been heavily tested is that articles or journals that are cited more often than 
others are deemed more important or relevant by the citing author. Therefore, the total 
number of citations a journal receives over a given period of time may be an indicator of 
its quality in relation to other journals. 

Many studies produce ranked lists of journals based on citation counts. The 
highest cited journals are presumed to be the most important. These ranked lists can 
establish journal “cores” 2 , reveal holes in subject collections, or provide a means to weed 
less relevant journals. Many have tested the validity of these types of rankings by 
comparing them to other data such as the total number of articles published by a journal 
within a discipline, journal circulation figures, library holdings, or inter-library loan 
requests. 

Another common method for verifying citation rankings is through expert opinion 
surveys. As a means of comparison, a group of individuals deemed experts by the 
researcher are asked to produce their own subjective list of journal rankings or are asked 
to verify the researcher’s list in some way. Some of these studies perform statistical tests 
to determine the degree of correlation between citation and expert opinion rankings. 
However, to this point, little research has been done to verify the precision of citation 
based rankings. 

To elaborate, other studies that test for rank order correlation typically do so to 

determine whether journals belong among the core journals. The purpose of these studies 

is generally to separate journals that are more important from those that are less important 

and therefore potentially suitable for de-selection. Consequently, beyond determining the 
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'• * overall accuracy of the journal rankings, these studies really focus only on the middle to 

lower ends of these lists. In fact, Line wrote in 1978 that “the precise rankings of the top 
journals is of no practical interest whatever.” 

Line’s statement may have been true in 1978, but the precision of these rankings 
is of particular interest now when librarians must decide which of these journals are to be 
offered in an electronic format. Therefore, this study will focus on only the top journals 
in a field. This study will not identify journals to be weeded but rather those to which 
access could be expanded. Considering it is often feasible to only offer the most 
important journals electronically, perhaps the top two or three, it is vital that these 
journals can be accurately identified. For instance, whether a journal ranked third on a 
list based on citations is in fact the third most relevant journal. This study will seek to 
support or refute the idea that citation rankings are an effective means of determining a 
journal’s importance relative to other journals. 

Purpose of the Study 

This paper is intended as an exploratory study to determine how precisely the positions of 
Political Science journals from a citation ranked list correspond to a list of the same 
journals ranked on a survey completed by a group of Political Science professors. In this 
fashion, it may be possible to support or refute citation analysis as a legitimate method of 
accurately ranking the most important journals in the field of Political Science. The 
major research questions explored in this study are: 

1) Can citation analysis accurately identify the highest quality journals? 

2) Can citation counts reliably distinguish between the quality of journals relative to 
one another in a ranked list? 

3) Do the results of citation analysis correlate with subjective measures? 
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r * If citation rankings correlate highly with subjective rankings, then citation analysis could 
be considered an effective tool to help choose journals to offer electronically. If it does 
not, it will be necessary to seek another kind of tool for that purpose. 

Limitations of the Study 

There are some important limitations to this study. First, not all citations are 
necessarily positive. Articles are sometimes cited specifically to challenge or refute 
findings or conclusions. Articles that are especially controversial may be cited far more 
often than others. In such cases, the greater number of citations would not necessarily 
indicate greater quality. However, since this study will focus on journals rather than 
individual articles, this will be less an issue. 

Another issue is that authors, either by choice or by neglect, do not always cite all 
the materials that they use. Articles that may legitimately serve as background material 
may not be cited in a final paper. Conversely, authors can cite articles from prestigious 
journals solely to confer a sense of legitimacy to their own work. Furthermore, authors 
often cite themselves or the work of their colleagues, inflating the relevance of past work. 
In this sense, what would seem at first glance, to be purely quantitative data may in fact 
be highly subjective. All the more reason to test the validity of citation data. 

There are also some important issues concerning the Social Sciences Citation 
Index from which all citation data for this study will be drawn. First, articles that are 
cited multiple times in a paper are only counted once. This means that a journal article, 
which may have contributed only peripherally to a study, will be counted the same as 
more relevant articles, thus conferring a greater degree of quality to the publishing 
journal than may be appropriate. Fortunately given that only one author is attributed to 
each article, articles with multiple authors are only counted once, avoiding potential 
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overemphasis of journals. However, different articles from the same author or same 
journal are appropriately counted as separate citations. 






Another very important issue regarding the citation data is that only journals 
included in the SSCI category “Political Science” will be considered. As long as a 
journal is listed in the Political Science category, it will be included in this study 
regardless of whether or not the journal is also included in another category. Due to this 
fact, some heavily interdisciplinary journals included in the Political Science category 
may in fact receive citations for articles that may not be truly relevant to the field of 
Political Science thus potentially skewing the results. Conversely, given the often multi- 
disciplinary nature of Political Science, it is likely that some professors polled will not 
find a specific journal listed on the survey because it was not listed as a Political Science 
journal by the SSCI. To account for this, there will be spaces provided on the distributed 
survey that will allow professors to include journals they feel belong among the top 
twenty most relevant to their work. 

Finally, due to issues of cost and time, the professors to be surveyed for this study 
have not been chosen randomly but rather as a matter of convenience. Though the 
professors chosen are believed to be generally representative of the field of Political 
Science, the restricted sample means that results of this study can not be generalized. 
Furthermore, since this study is restricted to Political Science journals, the citation 
analysis method may not be appropriate for all fields and again can not be generalized. . 
A more detailed description of the survey and its distribution will be made in the 
methodology chapter. 
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CHAPTER II. 



LITERATURE REVIEW 



When ISI’s first citation index, the Science Citation Index (SCI), was published 
annually for the first time in 1963, a number of studies were conducted to test the validity 
of the information it offered. A study conducted by ISI found that the papers of Nobel 
laureates were cited thirty times more frequently than the average for their field. Even 
after accounting for the higher rate of publication by these authors, the ISI study verified 
that papers written by Nobel laureates were cited at about twice the rate of their 
colleagues. 4 Approaching the subject from the opposite direction, ISI later found in a 
1977 study that highly cited scientists also tended to be honored scientists. In this study, 
over sixty percent of the 250 most highly cited scientists had also won the Nobel Prize or 
had been elected to at least one national academy of science. 5 Studies by K. E. Clark 6 , 
Bayer and Folger 7 , and by Virgo 8 , all found a correlation between citation frequency of 
individuals or specific papers and peer judgements of quality. 

Other studies have tested the validity of citation based judgements against other 
measures of journal quality. In a 1978 study, Goehlert hypothesized that highly cited 
journals were also likely to be heavily used journals. 9 Relying on citation data from a 
previous study 10 he found a significant correlation between the citation counts of the most 
highly cited Political Science and Economics journals and the number of times articles 
were requested from a specific journal by Political Science and Economics professors 
through his university’s document delivery service. He also found a correlation between 
the same citation data and data collected from a current content advisory service offered 
to the same professors. However, it is not sufficiently clear if the journals were used more 
frequently based on some subjective measure of their quality or if it was simply a fact 
that they were readily available from Goehlert’s library. 
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In a similar 1978 study, Pan compared journal citation counts to use counts in a 
specific library and found that highly cited journals were also highly used journals. 1 1 Pan 
compared citation data from the SCI to library use counts of the same journals based on 
circulation, interlibrary loan, and in house use figures. It is again not clear if Pan’s 
results were due to some judgement of quality or a factor of journal availability. 

In a 1983 study, Nisonger compared a list of highly cited Political Science 
journals to the holdings of five Washington D.C. area university libraries, presuming that 
respected research libraries would have the best serial collections and would therefore be 
more likely to hold the journals in question. He found that a significantly high 
percentage of the most highly cited journals were indeed held by these five libraries. 12 
However his citation data is a bit dubious because Nisonger based his study on citations 
pulled from only six Political Science journals. He also looked at only three years of 
citations for five of the journals and only one year in the case of the sixth. 

Many studies have tested for rank order correlation between lists based on citation 
counts and lists produced through expert opinion surveys. In a 1991 study, Kim finds a 
significant correlation between highly cited library and information science literature and 
the perceptions of library school deans and directors. Furthermore, Kim arrives at her 
set of “library science” journals in a rather clever way, choosing not to rely solely on the 
SSCI list of library science journals. However Kim acknowledges in her conclusion that 
further research is needed. It is important to note that her results can not necessarily be 
generalized for all academic disciplines. 

In another paper by Nisonger, he evaluates several earlier studies that test the 
correlation between citation and perception based measures and even conducts a study of 
his own. In all the studies at least a weak correlation between citation and perception 
measures is found and in some he finds a very high degree of correlation. 14 However, it 
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is not made clear in Nisonger’s article whether or not any of the correlation results were 
also tested for statistical significance. 

A high degree of correlation between citation rankings and subjective measures 
was also found in a 1982 study by Gordon 15 Gordon looked at the SSCI citation figures 
for sociology journals and polled 250 sociology professors drawn randomly from 
doctorate sociology programs in the U.S. He asked the professors to rank a list of sixty- 
three sociology journals against the American Sociology Review. He determined that his 
results were statistically significant but his method may not be appropriate for fields that 
do not have an appropriate journal to act as a benchmark. Gordon also noted a high 
degree of standard deviation in his ranking results. This indicated a general lack of 
agreement among the professors as to which journals were the best. 



Methodological Issues 

This study is similar to the work mentioned above in the sense that it too will test 
the correlation between a ranked list of journals based on citation counts and a ranked list 
derived from the survey of experts. However there are some important differences. First, 
most only focus on a two to three year period. A five-year time period will better account 
for yearly citation fluctuations and allow for all but the most recently published journals 
to be fairly counted against older and more established journals. Unfortunately, journals 
first published less than five years ago may not be fairly ranked against older journals. 

To account for this fact, spaces will be included on the survey so that professors may add 
journals they feel should be included among the top twenty. 

This study will focus on only the top twenty journal titles in a specific field, 
which will allow the precision of the rankings to be tested. Unlike previous work, this 
study will determine exactly how well the position of each individual journal correlates 
between the rank assigned to it by citation counting and the rank it is assigned by experts 
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Another important methodological issue is that this study will focus only on 
citation counts instead of a journal’s impact score. A journal’s “impact” is defined as 
“the average citation rate of a journal’s articles ,” 16 and is determined by the following 
ratio: 

For a specific journal, where 1999 = the current year. 

Journal X’s impact factor = 

Total number of citations made in 1999 to 1998 and 1997 articles appearing in journal X 
Number of citable items published by journal X in 1998 and 1997 

In other words, impact factor is the ratio determined by the number of times a journal’s 
articles printed in the previous two years were cited in the current year, divided by the 
total number of citable articles produced by that journal in the two previous years. The 
stated purpose of impact factors is to compensate, “for the advantage that older, larger, or 
more frequently published journals would enjoy if rankings were based on total citations 
received .” 17 

There are a number of papers both for and against the use of impact factors versus 
total citations. In Nisonger’s papers mentioned above and in one by Buffardi and 
Nichols 18 , the use of impact factors is strongly advised to account for the “advantage” 
that older or more prolific journals supposedly enjoy. The authors feel that impact scores 
make journal rankings fairer. However, in Pan’s paper, noted earlier, she finds no 
correlation between actual journal usage and impact factors, but does find a correlation 
between, “the rankings of journals by their citation and use counts .” 19 Though Gordon’s 
intent was not to promote one measure over another, he notes that “times cited correlates 
most highly with subjective indicators .” 20 Dometrius in fact writes in an article “fairness 
has nothing to do with journal stature.” How a journal may have attained its stature is 

irrelevant when comparing its citation rates to subjective judgements of its quality. This 
is not to say that impact factors may not be relevant when attempting to make distinctions 




9 



16 



of quality between journals with similar citation counts. More study is needed on this 
issue. 
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CHAPTER III. 



METHODOLOGY 

Collecting the data involved several steps. The first step was to establish a ranked 
list of Political Science journals based on citation counts. All data were pulled from the 
SSCI Journal Citation Reports. Each volume of the JCR contains annual citation counts 
for all journals indexed by the SSCI. Journals are also grouped by subject categories and 
this study focused solely on those classified in the Political Science category. The 
number of journals classified as Political Science varies a bit from year to year but 
includes approximately seventy titles. For the purpose of this study, citation data came 
from the most recent five volumes of the SSCI JCR which are the 1992-1996 editions. 

Each JCR from 1992-1996 was consulted for citation data. The total number of 
citations received by each Political Science journal for each year was added to the 
citations from the other years to form a five-year sum. A time period of this size 
produced a large enough sample of journals to minimize most citation aberrations and 
help account for relatively recent journal trends 

Any journal categorized at least once as a Political Science journal for the years in 
question was included and ranked against the other journals. The citation counts for any 
journal that had a name change over this five-year period had its total citations added to 
the citations for the newly named journal to form a unified total listed under a 
combination of both names. Using the combined citation data for each journal, the top 
twenty most cited titles were identified and noted in Appendix A. The titles were listed 
in order based on citation counts with the most cited journal listed first and so on. 
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Using these twenty journals, a survey was constructed and distributed to Political 
Science professors. (Appendix B) The survey was intended to test the perceived quality 
of the twenty most cited Political Science journals. Those individuals considered experts 
in a field, are often those who write and publish articles on topics within that field. 
Political Science professors are often expected and even required to publish original 
research in their field. Given the nature of the Political Science discipline, its professors 
should be very familiar with the journal literature and should make excellent judges of 
journal quality. The survey included instructions for its completion and return and asked 
the professors to identify their major area or areas of specialization. There were also 
additional spaces included on the survey so that the professors could add journals to the 
list that they felt should have been included among the top twenty. This was to account 
for the fact that some journals, perhaps those published only recently, might not be 
counted fairly against older journals. The survey was pre-tested on Penn State Political 
Science graduate students before a final survey was distributed. 

Surveys were sent to all the faculty members listed on the Political Science web 
sites of Penn State, Ohio State, and the University of Michigan. These schools were 
chosen based on their program descriptions included in their respective web sites. Their 
web sites indicated that they were large research-oriented programs that offered 
Doctorates in Political Science. The school's respective web sites also indicated each 
program employed a large number of professors with a wide variety of research interests. 
Large research-oriented schools are more likely to include professors from all the 
Political Science specialties. The specialties were established based on the nearly 
identical program concentrations listed for each school on their web pages and are 
identical to those identified by Nisonger 22 . The specialties are American Politics, 
International Relations, Comparative Politics, and Political Theory. With this 
information, it was possible to conduct additional tests for correlation within each 
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specialty. Surveys were also distributed to the faculty members of the Political Science 
department at Kent State with the hopes of a higher than average return rate. The total 
number of surveys sent was 122. 

The SPSS statistics and Microsoft Excel packages were used for data analysis. 
Each journal was placed in one of four general categories. The categories were the top 
one through five, top six through ten, top eleven through fifteen, and top sixteen to 
twenty. In addition, each professor was asked to specify which of the top five journals is 
the most important journal, the second most important journal, and so on until each of the 
top five journals was assigned to a specific rank. This effectively created a total of eight 
categories; one for each position in the top five and then the remaining three groupings. 
The results were aggregated for all professors as well as aggregated for specific subsets 
of professors based on their indicated specializations. 

Pearson’s r correlation coefficient was run for the total citations received by each 
journal and the number of times each journal was placed in the top one to five category. 
This statistic was appropriate because both total citations and the number of times each 
journal is placed in a top five position are interval level data. 

In addition, the average ordinal ranking for each journal was determined. Each 

journal had a number associated with it ranging from one to eight based on the eight 

categories. Journals placed in the top five were coded as one through five respectively, 

the next six through ten were coded as six, eleven through fifteen as seven, and sixteen 

through twenty as eight. As an example, the average ordinal ranking of a journal will be 

1.2 if forty of a total of fifty professors assign a journal a number one rank and the other 

ten professors assign it a number two rank.[ (40*1) + (10*2) / 50 = 1 .2 ] A numerically 

small average ranking such as this one would indicate a very high degree of perceived 

quality and a larger number, for instance a number very close to eight, would indicate a 

very low degree of perceived quality. In addition, the standard deviation for each journal 
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•' ‘ * was also calculated to indicate the level of agreement among the professors regarding a 

journal’s rank. A standard deviation will also be calculated for rankings within the four 
areas of specialization. 

Furthermore, a Spearman’s p ( rho ) correlation value was calculated using the 
ordinal rankings from one through twenty from the citation-based list and the average 
rank values from the journals derived from the survey data. This was repeated for each 
specialization as well. This statistic measures association for ordinal data. Each set of 
information was placed in a table along with the journal names. This stage of the 
analysis describes the results of the survey and establishes the rankings or relative 
importance of each of the journals in the set produced by the citation analysis. 

When analyzing the data, if the Pearson’s r coefficient and the Spearman’s p 
value are large and positive, this will indicate substantial agreement between the survey 
and citation methods. In other words, journals cited more frequently are, in fact, 
considered more important by researchers in the field. This result would support the 
validity of citation analysis as a tool for acquisition decisions. 

By comparing simple correlation and rank order correlation coefficients across 
each area of specialization it will be possible to determine whether or not citation analysis 
produces accurate ranked lists of journals for specific areas of the discipline. For 
example, if the r and p values for the specialization of Political Theory are substantially 
lower than the correlation measures for the other specializations, citation analysis would 
be a poor indicator of journal quality and a less useful tool for acquisition decisions in 
institutions which serve departments composed mainly of Political Theorists. 

Finally, the journals added by the professors in the survey and the number of 
times each new journal name is mentioned were identified. These data supported under 
what conditions citation analysis is unlikely to reflect the importance of journals in the 
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field. For example, if a particular journal, one that is new or highly specialized, is added 
to the list by a number of professors, this may indicate that citation analysis produced an 
incomplete set of core journals. Again, this would demonstrate the limits of citation 
analysis in estimating the importance of journals under specific circumstances. 



CHAPTER IV. 



ANALYSIS OF DATA 



A list of the twenty most highly cited journals from the Political Science category 
was derived from five years of SSCI JCR citation data and ranked according to the total 
number of citations. (Appendix A) This list of twenty journals was then arranged 
alphabetically to reduce bias and placed in a survey. (Appendix B) The survey was 
distributed to a total of 122 Political Science professors from Penn State University, Ohio 
State University, the University of Michigan, and Kent State University. Two weeks 
after the first mailing, a reminder post-card was sent to all the professors who had not yet 
responded. A second mailing of the survey was deemed unnecessary when the total 
number returned exceed forty, the minimum amount required. In all, a total of fifty-three 
surveys were returned making the average return rate about 43%. One survey could not 
be used due to the fact that none of the twenty journals had been ranked. 

Of the fifty-two remaining surveys, forty-eight professors chose only one area of 
specialization, one professor chose two, two professors did not indicate a specialization at 
all, and one professor wrote in an answer. In all, twenty-one professors indicated their 
specialization to be American Politics, thirteen indicated International Relations, nine 
indicated Comparative Politics, and six indicated their specialization to be Political 
Theory. The data from the surveys of the two professors who did not indicate a specialty 
and the one professor who wrote in an answer were included in the overall statistics but 
were not added to the statistics for the specific specialties. The data from the survey of 
the professor who chose two areas of specialization was included in the sets of statistics 
for the two specialties but only counted once in the general data. The fact that some 
specializations are represented more heavily than others may bias the general results to 



some degree. 
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In addition, not all surveys were completed correctly. This made it necessary to 
make certain assumptions when coding the data. First, despite instructions to the 
contrary, some respondents included the rankings of their write-in answers from the 
second page of the survey among the rankings from the first page. This resulted in 
instances where fewer than five journals were ranked in the top-five. To correct for this, 
it was necessary to collapse the rankings from these surveys. For instance, if a 
respondent included only three journals in the top-five and ranked them one, three, and 
five respectively, the journals were coded as one, two, and three. This is appropriate 
because the rankings are ordinal and the three journals are still the first, second, and third 
highest ranked journals among the twenty most cited journals. 

Also, some respondents did not rank all twenty journals. Therefore, it was 
assumed that unranked journals were also the journals that were least important to the 
respondents. Accordingly, journals that were not ranked were included in the lowest 
available category, the sixteen through twenty category. This is appropriate because 
excluding the unranked journals would have effectively created a ninth category. Given 
that the respondents who did rank all twenty journals appropriately were not offered this 
option, it would not be appropriate to assume that they would not have excluded journals 
as well. 

All data were coded based on these assumptions which resulted in a 
disproportionate number of lowly ranked journals. However this is really not much of a 
concern. If a journal’s subjective ranking does not correlate well with its citation ranking 
this would simply indicate a failure of citation analysis to predict accurately an expert’s 
subjective judgement of a journal’s quality. 




17 



24 



Table L Measures of Association Between Citation Statistics and Expert Opinion 

Measures of Journal Evaluation 



Faculty Specialties 


Pearson’s r 


Spearman’s p ? (rho) 


All Faculty 


.73* 


.11 


American Politics 


.69* 


.12 


Comparative Politics 


.52* 


.01 


International Relations 


.77* 


.16 


Political Theory 


.52* 


.11 



Note: N=20 for all reported analyses. 

*p<.05 (two-tailed test) 

^Pearson correlation coefficients reflect the association between the total number of citations received by 
the twenty most-cited Political Science journals and the number of top five rankings the journals were 
assigned by survey respondents. 

^Spearman correlation coefficients reflect the association between citation rankings of the twenty most- 
cited Political Science journals and the average ranks assigned to the journals by the survey respondents. 



With all data entered, the first stage of analysis was to calculate a Pearson’s r 
coefficient. The Pearson’s r coefficient is a “measure of how strongly the x and y values” 
in a sample are related and is especially useful for interval level data. In this case the x 
value is the total number of citations each journal received and the y value is the total 
number of times each journal was ranked in the top-five by the survey respondents. This 
tests the assumption that if a journal is highly cited it is also likely to be ranked heavily in 
the top-five and if the journal is cited less often it will not be ranked as highly. In Table 
1, the large and positive coefficients indicate a high degree of correlation between the 
two measures. The degree of correlation varies somewhat within specializations but 
remains large and positive in all cases. 

These results would seem to support the idea that a high number of citations may 
in fact be an accurate predictor of a journal’s subjective ranking. As a means to verify 
these results, ordinal data was also collected for each journal. A Spearman’s rank 
correlation coefficient also called Spearman’s p ( rho ), tests for association between 
ordinal data. A Spearman’s p was calculated between the ordinal ranking based on total 
number of citations received and the average ranking assigned to each journal by the 
survey respondents. One would expect another large and positive result from this 
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measure. However, as indicated by Table 1, this relationship, though positive, is in fact 
very small and is not statistically significant, (p > .05) Give the disparity between the 
two measures, it is appropriate to reexamine the Pearson’s r results. 

The problem with Pearson’s correlation coefficient is that the value of r “ can be 
greatly affected by the presence of even one or two outlying ( x,y ) pairs.” 24 The 
Spearman’s correlation coefficient is a measure, “that is not as sensitive as r to outlying 
points.” 25 So the question is, is the disparity between the two measurements a result of 
one or two journals exerting undue influence on the remaining journals? 

A scatter-plot graph (Figure 1) shows the twenty journals plotted by their citation 
counts (x) and the number of top-five rankings received by each (y). The three journals 
that received the highest number of top-five rankings, the American Political Science 
Review (APSR), the American Journal of Political Science (AJPS), and the Journal of 
Politics (JOP) have been labeled to provide points of reference. The line on the graph is 
called the “line of best fit” or simply the best-fit line. It is drawn to minimize the squared 
distance from any of the points and the line. If the relationship between the two variables 
x and y were perfect, each point would fall directly on the line. The line acts to visually 
illustrate the degree of correlation between the two variables. 

Judging from Figure 1, it certainly seems possible that the APSR and even 
perhaps the AJPS could in fact be exerting undue influence on the value of r. To assess 
the impact of “a point whose x value differs greatly from others,” it is appropriate to 
“delete it from the data set and then recompute the best-fit line and various other 
quantities.” 26 Figure 2 and Figure 3 are included to show the results of this process. 
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Figure 1. Citation Count and Top Five Rankings, N=20 
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Figure 2. Citation Count and Top Five Rankings, N=19 
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Figure 3. Citation Count and Top Five Rankings, N=18 
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Figure 2 is the same graph with the APSR excluded and the best-fit line 
recomputed. It clearly shows a weaker positive relationship between x and jy than seen in 
Figure 1. Figure 3 excludes both the APSR and the AJPS. The recomputed best-fit line 
shows a very weak positive relationship between x and y. It would seem that these two 
journals are in indeed exerting undue influence on the value of r. 

These figures are not intended to dispute the fact that these two journals are both 
highly cited and highly ranked. It is simply that these two journals received so many 
more citations than average and so many more top-five rankings that they created an 
impression that a relationship existed where in fact it may not. The average number of 
citations for the twenty most cited journals is 2774 and the median number of citations is 
only 1936. The APSR and AJPS received 12,706 and 5720 citations respectively. The 
average number of top-five rankings is 10.75 and the median number of top-five rankings 
is 5.5. The APSR and AJPS received 47 and 39 top-five rankings respectively. By 
eliminating these influential observations, it helps to illustrate the real relationship 
between the number of citations received and the number of top-five rankings. 



Table 2. Pearson Correlation^ Coefficients for Citation Counts and Number of 



Top Five Rankings Excluding Influential Observations 



Faculty Specialties 


All Journals 
(N=20) 


Excluding 
One Journal 1 
(N=19) 


Excluding 
Two Journals^ 
(N=18) 


All Faculty 


.73* 


.46* 


.05 


American Politics 


.69* 


.42 


.07 


Comparative Politics 


.52* 


.24 


-.09 


International Relations 


.77* 


.60' 


.27 


Political Theory 


.52* 


.20 


-.15 



p<.05 (two-tailed test) 

Vearson’s correlation coefficients reflect the association between the total number of citations received by 
the twenty most-cited Political Science journals and the number of top five rankings the journals were 
assigned by survey respondents. 

* Excludes a single influential observation, American Political Science Review. 
n Excludes two influential observations, American Political Science Review and American Journal of 
Political Science. 
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Table 2 shows that when r is recomputed after excluding the two journals in 
question, there is no statistically significant relationship between the number of citations 
and the number of top-five rankings. This result tends to support the results of the prior 
Spearman’s rank correlation. It is still possible however, at least in the case of the APSR 
and the AJPS, that citation analysis did in fact correctly identify the two "best" Political 
Science journals. Considering the intent of this study is to determine whether or not the 
very best Political Science journals can be accurately identified, a resulting set of two 
journals is not necessarily inappropriate. The APSR received by far the most citations 
and number of top-five rankings than any other journal and the AJPS received a number 
of citations and top-five rankings second only to the AJPS. Could it therefore be 
reasonably assumed that these two journals are in fact number-one and number-two 
respectively in terms of quality? 



Table 3. Average Rank Assigned by Survey Respondents Including the Standard 
Deviation for the Five Most-Cited Political Science Journals 









Faculty Specialties 




All 

Faculty 


American 

Politics 


Comparative 

Politics 


International 

Relations 


Political 

Theory 


American Political 


1.98 


1.71 


2.67 


2.08 


1.33 


Science Review 


(1.86) 


(1.38) 


(2.60) 


(2.25) 


(0.52) 


American Journal of 


2.90 


1.80 


4.22 


3.14 


3.83 


Political Science 


(1.79) 


(0.62) 


(2.39) 


(1.79) 


(1.83) 


Economic and 


7.86 


7.95 


7.75 


7.77 


7.83 


Political Weekly 


(0.41) 


(0.22) 


(0.46) 


(0.60) 


(0.41) 


Journal of Politics 


4.28 


3.43 


5.57 


4.85 


3.83 




(1.41) 


(1.03) 


(1.13) 


(1.34) 


(1.33) 


Journal of Conflict 


6.18 


6.75 


7.11 


4.62 


6.50 


Resolution 


(1.56) 


(0.72) 


(0.78) 


(2.06) 


(0.55) 




N=52 


N=21 


N=9 


N=13 


N=6 



Note: Table entries are mean rankings assigned to journals by survey respondents. Standard deviation of 
rankings is in parentheses. 
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If such an assumption were valid, the level of agreement among the survey 
respondents regarding the two journal's average ranking should be very high. However, 
the large standard deviation figure for both journals illustrated in Table 3 indicates that 
while most professors do indeed agree that both the APSR and the AJPS belong among 
the top-five journals, they do not agree as to which positions in the top-five the two 
journals actually belong. Therefore we can not reasonably assume that the APSR and 
AJPS are the first and second best Political Science journals respectively. 

There are two other interesting observations to be made from Table 3. First, the 
journal with the lowest standard deviation, Economic and Political Weekly, is also the 
journal with the lowest 27 average ranking among the five most cited journals. In fact, 
despite receiving the third highest number of citations overall, Economic and Political 
Weekly was ranked as the least important among the twenty most cited journals. 
(Appendix C) Like Economic and Political Weekly, the three other journals with the 
lowest average ranking also have a relatively high number of citations. So why are these 
journals ranked so poorly despite a high number of citations? 

Some of the survey respondents offer a possible answer. Notes written on three 
separate surveys each indicate that Economic and Political Weekly, the Nation, the New 
Left Review, and the New Republic are in fact magazines, not journals. What 
distinguishes a magazine from a journal is not immediately clear and is not an issue 
explored in this paper but as one survey respondent notes, none of the four periodicals are 
peer-reviewed. Unfortunately, there is insufficient information to explain this apparent 
failure of citation analysis. This matter certainly warrants future investigation. 

The second point of interest illustrated in Table 3 and which is even more 

apparent in Appendix C is the fact that the standard deviation for each journal almost 

always tends to be lower within the specializations than among the faculty in general. 

This may indicate that a survey respondent’s area of specialization significantly 
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influences the rank they assign each journal. Is it possible that the apparent lack of 
correlation between a high citation count and high subjective ranking for Political 
Science journals is due to a high level of disagreement among the four specializations? 
Conversely, can it be assumed that a high level of agreement will always exist within 
homogeneous specializations? Or is it possible that citation counts alone are insufficient 
to distinguish the twenty most important Political Science journals? 

If the twenty journals were accurate and the lack of correlation was due simply to 
a high level of disagreement among the specializations, one would expect relatively few 
write-in answers from the survey. As indicated by Appendix D, the very large number of 
write-in answers would seem to indicate that this is not the case. In addition, the fact that 
only about half of the respondents chose to write in a journal name does not necessarily 
indicate the respondents who chose not to write in an answer were in complete agreement 
with the list of journals. Because the survey did not allow the respondents to exclude any 
journals from the top twenty, it is not clear if the respondents who failed to rank certain 
journals did so because they felt the journals did not belong among the top twenty. 



Table 4. Most Frequently Mentioned Write-In Journals 





Number of 
Mentions* 


Number of 
Top-Five Rankings* 


World Politics 


19 


14 


International Studies Quarterly 


14 


6 


International Organization 


12 


7 


Public Opinion Quarterly 


9 


2 


Polity 


7 


2 


International Security 


4 


1 


Political Analysis 


4 


2 



Respondents were asked to write in any journals that they felt should have been included among the top 
twenty political science journals. Number of mentions is the total number of times the journal was added 
by survey respondents. 

^Respondents were also asked to rank their write-in journals. Number of top-five rankings is the total 
number of times respondents ranked a write-in journal among the top five political science journals. 
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It seems clear from Table 4 that a number of journals may have been 
inappropriately excluded from the list of top twenty Political Science journals. So why 
were these journals and the other journals listed in Appendix D excluded? The simple 
answer for some is that they just did not receive as many citations as the other journals. 
For instance Polity, listed above, received only 522 total citations from 1992-1996. The 
fact that a number of respondents wrote in Polity may indicate some respondents use the 
journal more often than average. 

This author does not know whether the articles from Polity typically focus on a 
specific specialization within the Political Science discipline. If this is indeed the case, 
this may explain its exclusion from the top-twenty most cited journals but its inclusion by 
some survey respondents. It is interesting to note in fact that four of the seven survey 
respondents who wrote in Polity identified themselves as political theorists. Given that 
within this entire study, there were only six respondents who identified themselves as 
political theorists, this indicates two-thirds of them agreed that it should have been 
included. 

There is also a potentially more serious problem suggested by Table 4. The 
journals World Politics, International Studies Quarterly, International Organization, 
International Security, and Public Opinion Quarterly were not included in the SSCI 
Political Science category and were therefore precluded from this study. Public Opinion 
Quarterly is listed in the SSCI within the Communications category and the other four 
are all included in the International Relations category. It is not clear why the SSCI did 
not cross-list these journals in both the Political Science category and the 
Communications and International Relations categories respectively. Some journals such 
as the Journal of Conflict Resolution are indeed listed in both the Political Science and 
International Relations categories 
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The problem is that the solution is not as easy as simply including all the journals 
listed as Communications or International Relations within a future survey. At first 
glance it would seem obvious to include the International Relations journals with the 
Political Science journals given that a major Political Science specialization is 
International Relations. The issue is that though the two may share the same name, the 
SSCI International Relations category does not necessarily denote the same area of study 
as the Political Science area of specialization. 

The SSCI category is in fact very multi-disciplinary. Judging from the titles 
listed, there seem to be a large number of legal, business, and economics journals 
included in addition to a number of Political Science journals. It may be that these 
journals are indeed of interest to political scientists but including the citation counts from 
these journals could potentially skew a citation ranking. It is simply not possible to 
determine who cited a particular journal or why. The same can be said of course of 
Political Science journals but at least in their case the topic tends to remain a legitimate 
Political Science topic from article to article. 

The solution to this dilemma is not clear. If future work is to be done in this area 
it will be necessary to come up with a more inclusive list of Political Science journals. 
This is especially true given that journal titles such as World Politics and Public Opinion 
Quarterly are very heavily cited and would certainly have been included within a list of 
the twenty most cited Political Science journals. The exclusion of these tvvo journals as 
well as many of the others is really a failure of the citation analysis method used in this 
study and does not necessarily indicate a failure of citation analysis in general. However, 
the one journal that has not been discussed from Table 4, Political Analysis, may very 
well have been excluded because of an inherent weakness in citation analysis. 

Political Analysis is a very new journal and is not even listed in the 1996 edition 
of the SSCI JCR. In addition, according to one of the survey respondents, Political 
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Analysis is currently published on an annual basis only, a fact that is apparently soon to 
change. The respondent is so convinced of the journal’s quality that he/she writes that it 
will become “a top five guaranteed.” This is an excellent example of a limitation of 
citation analysis. Clearly in some cases, fewer citations is not necessarily an indication of 
lesser quality. Future studies must account for this problem or risk producing results that 
are not accurate. 
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CHAPTER V. 



CONCLUSION 

Based on the results of this study, it seems that in the case of Political Science 
journals, citation counts alone are not sufficient to correctly identify the "best" journals. 
The initially large and positive Pearson's correlation coefficient was refuted when it 
became apparent that the American Political Science Review and the American Journal of 
Political Science were acting as influential observations. The results of the Spearman's 
rank order correlation would seem to support these findings. Given that a rank order 
correlation coefficient is less influenced by a few atypical results very far from the mean, 
this conclusion seems appropriate. 

In addition, given the high degree of standard deviation for the APSR and the 
AJPS it seems the survey even failed to confirm the possibility that those two journals are 
in fact the best two Political Science journals. Though the results support both journal's 
ranking among the top-five most important Political Science journals, no further 
conclusions are possible. In fact the standard deviation figures suggest that the highest 
level of agreement occurs for journals that perhaps should have been excluded from the 
survey altogether. In most other cases, the standard deviation is only relatively low 
within the four major Political Science specializations and in some cases not even then. 

It also seems given the very large number of journal names added by survey 
respondents, that the citation analysis method used for this study may have failed to even 
correctly identify the twenty best Political Science journals let alone rank them 
accurately. Some journals such as World Politics and Public Opinion Quarterly are 
mentioned with such frequency and are assigned such a high ranking, that no other 
reasonable conclusion can be reached. 
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However, given that this study relied on the rather narrow definition of a Political 
Science journal offered in the Social Science Citation Index Journal Citation Reports, the 
citation analysis method used for this study may be inherently flawed. With this fact in 
mind, it can not be argued conclusively that citation counts can not identify at least some 
aspect of a journal's quality. It appears likely however that citation counts alone are 
simply not a sufficient measure to rank accurately a journal's quality. 

The need for an accurate process to identify the "best" journals still exists. As 
patrons demand greater access to information this need will become increasingly more 
acute. Future studies devoted to this problem will also have to contend with the problem 
of specializations with a field of study. 

It seems reasonable to assume that even within a given discipline, professors with 
different specializations within that discipline may have legitimately different views of 
which journals are the most important. Beyond a select few journals that may include 
work on all the major specialties of a discipline, the APSR and the AJPS for example, 
other journals deemed more important by specific professors are likely to fall within their 
respective areas of specialization. 

Unless a citation analysis method is devised that accounts for this issue of 
specialization, citation analysis may prove to of little practical value when trying to 
identify only the most important journals in a field of study. It may be the case that 
citation analysis simply can not provide better than an educated guess as to a journal's 
quality. 

This study fails to identify an adequate method of identifying and ranking a 
journal's quality. Given the diverse interests of professors, it may only be practical to 
identify the specific needs of faculty on a school to school basis. Attempting to 
accurately identify the "best" journals on a large scale may prove difficult at best. With 
additional issues beyond those discussed in this study such as cost, quality of access, and 
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the type of educational institution, the decisions facing many librarians are unenviable. 
There clearly remains a need for better tools to aid the selection process especially when 
it comes to choosing electronic journals. More extensive and timely research is needed 
on this matter. 
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Appendix A 



Twenty Most Frequently Cited Political Science Journals 





Citation 

Count* 


Citation 

Rank 


Total Top- 
Five 

Rankings* 


Average 

Rank** 


Standard 

Deviation** 


American Political Science Review 


12706 


1 


47 


1.98 


1.86 


American Journal of Political Science 


5720 


2 


39 


2.90 


1.79 


Economic and Political Weekly 


3858 


3 


1 


7.86 


0.41 


Journal of Politics 


3683 


4 


31 


4.28 


1.41 


Journal of Conflict Resolution 


3678 


5 


6 


6.18 


1.56 


Public Choice 


3656 


6 ' 


4 


6.94 


1.16 


New Republic 


2902 


7 


1 


7.80 


0.41 


Annals of the American Academy of 
Political and Social Sciences 


2446 


8 


2 


7.27 


0.78 


New Left Review 


2360 


9 


1 


7.63 


1.08 


Nation 


2132 


10 


0 


7.67 


0.59 


British Journal of Political Science 


1740 


11 


19 


5.64 


1.52 


Legislative Studies Quarterly 


1402 


12 


5 


6.66 


1.17 


European Journal of Political Science 


1320 


13 


2 


7.12 


1.01 


Political Science Quarterly 


1265 


14 


7 


6.38 


1.03 


Comparative Politics 


1227 


15 


17 


5.53 


1.94 


Political Studies London 


1187 


16 


3 


7.28 


0.97 


Comparative Political Studies 


1122 


17 


10 


6.02 


1.68 


Political Geography 


1106 


18 


3 


7.38 


1.03 


American Politics Quarterly 


1013 


19 


9 


6.44 


1.11 


Political Theory 


954 


20 


8 


6.29 


1.88 



Note: N=52 for all analyses. 

Citation count, derived from Social Science Citation Index Journal Citation Reports, reflects total number of citations 
to journal from 1992 through 1996. 

*Total number of top-five rankings indicates the total number of times the survey respondents ranked a journal in the 
top-five. 

n Mean rank assigned to the journal by the survey respondents. 

^Standard deviation of the ranks assigned to the journal by the survey respondents. 
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Appendix B 



School of Library and Information Science 

(330) 672-2782 
Fax (330) 672-7965 
http'J/www. slis. kent. edu 




Re: Testing for Correlation Between Two Journal Ranking Methods: A Comparison of 
Citation Rankings and Expert Opinion Rankings. 

March 8, 1999 

Dear Professor «last»: 

I am a graduate student in the School of Library and Information Science at Kent State 
University. As part of the requirements for my Master's degree I am conducting a study about the 
accuracy of periodical rankings based on citation counts. Some librarians, seeking guidance for 
collection development decisions, have turned to citation analysis methods to assist them. 

The citation analysis method focuses on citations from journal articles and other sources. The 
method works by discovering patterns within these citations. It has been supported in earlier 
studies that the number of times a specific journal is cited over a given period of time can indicate 
the overall "quality," "importance," or perhaps "relevance" of that journal. I am testing the degree 
of correlation between citation based rankings and the subjective rankings of experts. If the 
correlation is very high, this could indicate that libraries might be able to rely on citation data to 
help with acquisition decisions such as which journals would be best to offer electronically. As 
an expert in Political Science, you are being asked to participate in this study. 

Confidentiality is guaranteed, as you do not need to sign your name to individual questionnaires; 
only I, as the investigator, have access to the survey data. There is no penalty of any kind if you 
choose not to participate in this survey. While your cooperation is essential to the success of this 
study, it is, of course, voluntary. A copy of the results of the study will be available upon request. 

If you have any further questions, please contact me at (814) 237-2278 or contact Dr. Richard 
Rubin, my research advisor, at (216) 672-2782. If you have any further questions regarding 
research at Kent State University you may contact Dr. M. Thomas Jones, at (216) 672-2851 . 
Thank you very much for your cooperation; it is much appreciated. You may return the 
questionnaire in the enclosed self-addressed, stamped envelope to me at the following address: 

Robert Russell (department mailbox # 255) 

c/o School of Library and Information Science 

P.O. Box 5 190 

314 University Library 

Kent State University 

Kent OH, 44242-0001 

Sincerely, 



Robert Russell 



«codel»«code2» 



Citation Rank Verification Survey 



Page 1 of 2 



Please complete both pages of this survey. Your responses to this survey are confidential. 



Please indicate the area of political science in which you conduct the majority 
of your research and/or teaching. Feel free to choose as many as apply. 

American Politics Comparative Politics 

International Relations Political Theory 



Please rank all twenty journals below based on your perception of their quality. For the journals you feel 
belong among the top five best journals, please rank them according to their quality starting with the 
number!, indicating the best journal, and continuing to rank the second, third, fourth and fifth best journals 
with the numbers 2, 3, 4, and 5 respectively. Please indicate with only a check mark , if you feel the 
remaining fifteen journals belong among the top 6-10 journals, the top 1 1-15 journals, or the top 16-20 
journals. Please include only five journals in each category. You may add journals on the next page that 
you feel should have been included among the top twenty. 



Journal Name Rank the Check the Check the Check the 

Top 1-5 Next 6-10 Next 11-15 Next 16-20 

American Journal of Political Science 

American Political Quarterly 

American Political Science Review 

Annals of the American Academy 

of Political and Social Sciences 

British Journal of Political Science 

Comparative Political Studies 

Comparative Politics 

Economic and Political Weekly 

European Journal of Political Research 

Journal of Conflict Resolution 

Journal of Politics 

Legislative Studies Quarterly 

Nation 

New Left Review 

New Republic 

Political Geography 

Political Science Quarterly 

Political Studies - London 

Political Theory 

Public Choice 



Survey Continues on Next Page. . . 



Page 2 of 2 



Please write in and rank as above any joumal(s) you feel should have been included on this survey. Please 
do not alter the rankings from the prior section to reflect the additions made below. Feel free to include as 
many journal names as you feel are appropriate and use more space if necessary. 



Next 16-20 



Journal Name 



Rank the Top 1-5 Check the Next 6-10 Next 11-15 



Thank you for completing this questionnaire. Your time and effort are greatly appreciated. Mailing 
instructions are included in the cover letter mailed with this survey. If you would like to receive a summary 
of the results of this survey, please check the box below: 

Yes, I would like to receive a summary of this report upon its completion. 
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Appendix C 



Average Rank Assigned by Survey Respondents Including 
the Standard Deviation 



Faculty Specialties 





All 

Faculty 


American 

Politics 


Comparative 

Politics 


International 

Relations 


Political 

Theory 


American Political 


1.98 


1.71 


2.67 


2.08 


1.33 


Science Review 


(1.86) 


(1.38) 


(2.60) 


(2.25) 


(0.52) 


American Journal of 


2.90 


1.8 


4.22 


3.14 


3.83 


Political Science 


(1.79) 


(0.62) 


(2.39) 


(1.79) 


(1.83) 


Economic and Political 


7.86 


7.95 


7.75 


7.77 


7.83 


Weekly 


(0.41) 


(0.22) 


(0.46) 


(0.60) 


(0.41) 


Journal of Politics 


4.28 


3.43 


5.57 


4.85 


3.83 




(1.41) 


(1.03) 


(1.13) 


(1.34) 


(1.33) 


Journal of Conflict 


6.18 


6.75 


7.11 


4.62 


6.50 


Resolution 


(1.56) 


(0.72) 


(0.78) 


(2.06) 


(0.55) 


Public Choice 


6.94 


6.75 


7.22 


7.38 


6.83 




(1.16) 


(1.07) 


(0.83) 


(0.77) 


(0.75) 


New Republic 


7.80 


7.85 


7.78 


7.92 


7.33 




(0.41) 


(0.37) 


(0.44) 


(0.28) 


(0.52) 


Annals of the American 


7.27 


6.90 


7.63 


7.62 


7.17 


Academy of Political 
and Social Sciences 


(0.78) 


(0.83) 


(0.52) 


(0.51) 


(0.98) 


New Left Review 


7.63 


7.90 


7.67 


7.38 


7.00 




(1.08) 


(0.30) 


(0.71) 


(1.94) 


(0.63) 


Nation 


7.67 


7.81 


7.44 


7.77 


7.33 




(0.59) 


(0.51) 


(0.88) 


(0.44) 


(0.52) 


British Journal of 


5.64 


5.39 


5.22 


6.38 


5.50 


Political Science 


(1.52) 


(1.37) 


(1.39) 


(1.94) 


(1.22) 




N=52 


N=21 


N=9 


N=13 


N=6 



Note: Table entries are mean rankings assigned to journals by survey respondents. Standard deviation of rankings is in 
parentheses. {continued) 
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Average Rank Assigned by Survey Respondents Including 
the Standard Deviation ( continued 





All 

Faculty 




Faculty Specialties 




American 

Politics 


Comparative 

Politics 


International 

Relations 


Political 

Theory 


Legislative Studies 


6.66 


5.7 


7.22 


7.62 


7.00 


Quarterly 


(1.17) 


(0.86) 


(0.97) 


(0.77) 


(0.89) 


European Journal of 


7.12 


7.29 


6.67 


7.00 


7.67 


Political Science 


(1.01) 


(0.78) 


(1.12) 


(1.35) 


(0.52) 


Political Science 


6.38 


6.24 


6.89 


6.85 


5.50 


Quarterly 


(1.03) 


(0.89) 


(1.05) 


(0.90) 


(0.55) 


Comparative Politics 


5.53 


6.10 


3.88 


6.00 


5.83 




(1.94) 


(1.84) 


(2.23) 


(1.35) 


(0.98) 


Political Studies London 


7.28 


7.29 


7.33 


7.77 


6.20 




(0.97) 


(0.85) 


(0.87) 


(0.44) 


(1.79) 


Comparative Political 


6.02 


6.57 


4.11 


5.92 


7.17 


Studies 


(1.68) 


(1.21) 


(1.90) 


(1.61) 


(0.98) 


Political Geography 


7.38 


7.57 


7.67 


6.79 


• 7.83 




(1.03) 


(0.60) 


* (0.71) 


(1.63) 


(0.41) 


American Politics 


6.44 


5.67 


7.22 


7.08 


6.50 


Quarterly 


(1.11) 


(0.86) 


(0.97) 


(0.76) 


(1.22) 


Political Theory 


6.29 


6.86 


6.89 


7.08 


2.17 




(1.88) 


(1.01) 


(0.93) 


(0.95) 


(1.17) 




N=52 


N=21 


N=9 


■ N=13 


N=6 



Note: Table entries are mean rankings assigned to journals by survey respondents. Standard deviation of rankings is in 
parentheses. 




40 50 



Appendix D 



Write-In Journals 



Number of Number of 

Mentions* Top-Five Rankings* 


Asian Survey 


2 


1 


Australian Journal of Political Science 


1 


1 


Behavioral Science 


1 


0 


Daedelus 


1 


1 


Diacritics 


1 


1 


Differences 


1 


0 


Dissent 


1 


1 


Economics and Politics 


2 


0 


Economics and Society 


1 


1 


Economist 


1 


1 


Electoral Studies 


1 


0 


Environment and Planning 


1 


0 


European Journal of International Relations 


1 


1 


Feminist Studies 


1 


1 


Foreign Affairs 


3 


2 


Foreign Policy 


3 


1 


Government and Opposition 


1 


1 


Hypatia: Journal of Feminist Philosophy 


2 


1 


International Interactions 


3 


0 


International Journal of Public Opinion Research 


1 


1 


International Organization 


12 


7 


International Political Science Review 


1 


0 


International Security 


4 


1 


International Studies Quarterly 


14 


6 


Journal of Commonwealth and Corporate Studies 


1 


1 


Journal of Conflict Management and Peace Science 


2 


0 


Journal of Economic Perspectives 


1 


0 


Journal of Peace Research 


2 


0 


Journal of Policy Analysis and Management 


1 


2 


Journal of Political Economy 


1 


0 


Journal of Theoretical Politics 


3 


0 


Latin American Research Review 


1 


0 


National Journal 


1 


1 


New German Critique 


2 


1 


Pacific Affairs 


1 


1 


Policy Studies Journal 


3 


1 


Political Analysis 


4 


2 


Political Behavior 


3 


1 


Political Psychology 


2 


0 



(continued) 
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Write-In Journals (continued) 



Number of Number of 

Mentions t Top-Five Rankings' 1 ' 


Political Research Quarterly 


3 


1 


Politics and Society 


1 


0 


Polity 


7 


2 


Presidential Studies Quarterly 


1 


0 


Public Administration and Development 


1 


1 


Public Administration Review 


3 


2 


Public Culture 


1 


0 


Public Opinion Quarterly 


9 


2 


Publius: The Journal of Federalism 


2 


1 


Review of International Political Economy 


1 


0 


Review of Politics 


1 


0 


Security Studies 


1 


0 


Signs 


2 


1 


Social Science Quarterly 


1 


1 


Theory and Event 


2 


1 


Theory and Society 


1 


0 


West European Politics 


1 


1 


Western Political Quarterly 


1 


0 


World Politics 


19 


14 



Respondents were asked to write in any journals that they felt should have been included among the top 
twenty Political Science journals. Number of mentions is the total number of times the journal was added 
by survey respondents. 

Respondents were also asked to rank their write-in journals. Number of top-five rankings is the total 
number of times respondents ranked a write-in journal among the top five Political Science journals. 
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ABSTRACT: 



This study tests for correlation between two journal ranking methods, citation 
rankings and expert opinion surveys. Political Science professors from four major 
universities were asked to rank a list of the twenty most highly cited Political Science 
journals. Citation datai were taken from the Social Science Citation Index Journal 
Citation Reports from 1992-1996. In addition, each professor was asked to identify his 
or her area of specialization. A Pearson's correlation coefficient and a Spearman's rank 
order correlation coefficient were calculated from the survey data. In addition the 
standard deviation for the average ranking of each journal was calculated to show the 
level of agreement among survey respondents. Each calculation was repeated within 
each area of specialization. 

The resulting data did not indicate a statistically significant degree of correlation 
between citation counts and expert opinion surveys. However, there was some indication 
that a significant level of agreement did exist within each area of specialization. The 
results suggest that further research is necessary to determine how much effect 
specializations have on subjective journal rankings. 
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